from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.015	0.056	0.000	0.002	-1	1	0.676	20.115	0.082	0.676	0.100	0.100	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.000	0.026	-1	1	0.000	0.360	0.009	0.000	0.072	0.072	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.109	0.030	0.000	0.003	-1	5	0.743	20.016	0.025	0.743	0.155	0.155	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.003	0.000	0.028	-1	5	1.000	0.358	0.010	1.000	0.078	0.078	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.386	0.006	0.000	0.002	1	100	0.846	20.094	0.062	0.846	0.119	0.119	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.000	0.000	0.023	1	100	1.000	0.357	0.012	1.000	0.064	0.064	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.136	0.052	0.000	0.003	-1	100	0.846	20.141	0.109	0.846	0.156	0.156	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.000	0.026	-1	100	1.000	0.359	0.006	1.000	0.071	0.071	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.348	0.011	0.000	0.002	1	5	0.743	19.934	0.028	0.743	0.118	0.118	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	1	5	1.000	0.347	0.005	1.000	0.065	0.065	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.239	0.008	0.001	0.001	1	1	0.676	20.424	0.038	0.676	0.061	0.061	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.001	0.000	0.022	1	1	0.000	0.346	0.005	0.000	0.063	0.063	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.782	0.028	0.000	0.002	-1	1	0.845	4.329	0.016	0.845	0.412	0.412	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1	2	0.009	0.003	0.000	0.009	-1	1	1.000	0.287	0.005	1.000	0.032	0.032	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.031	0.041	0.000	0.003	-1	5	0.883	4.249	0.011	0.883	0.713	0.713	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	2	0.009	0.002	0.000	0.009	-1	5	1.000	0.287	0.005	1.000	0.033	0.033	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.314	0.004	0.000	0.002	1	100	0.887	4.329	0.016	0.887	0.535	0.535	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	1	100	1.000	0.285	0.005	1.000	0.012	0.012	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.026	0.031	0.000	0.003	-1	100	0.887	4.316	0.014	0.887	0.701	0.701	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	2	0.011	0.002	0.000	0.011	-1	100	1.000	0.288	0.004	1.000	0.039	0.039	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.319	0.004	0.000	0.002	1	5	0.883	4.259	0.011	0.883	0.544	0.544	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	1	5	1.000	0.288	0.004	1.000	0.012	0.012	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.096	0.003	0.000	0.001	1	1	0.845	4.328	0.007	0.845	0.253	0.253	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.285	0.005	1.000	0.007	0.007	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.727	0.008	0.000	0.002	1	5	0.975	127.545	0.000	0.975	0.014	0.014	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	1.000	3.057	0.043	1.000	0.001	0.001	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.944	0.017	0.000	0.001	-1	5	0.975	128.602	0.000	0.975	0.007	0.007	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.002	0.000	0.005	-1	5	1.000	3.050	0.041	1.000	0.001	0.001	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.494	0.004	0.000	0.000	-1	1	0.964	133.698	0.000	0.964	0.004	0.004	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	1.000	2.975	0.062	1.000	0.001	0.001	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.806	0.005	0.000	0.001	1	1	0.964	135.395	0.000	0.964	0.006	0.006	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	1.000	2.964	0.073	1.000	0.000	0.000	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.153	0.021	0.000	0.005	1	100	0.973	132.287	0.000	0.973	0.039	0.039	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	1.000	2.959	0.048	1.000	0.001	0.001	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.009	0.016	0.000	0.003	-1	100	0.973	136.798	0.000	0.973	0.022	0.022	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	1.000	2.989	0.079	1.000	0.002	0.002	See	See
12	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.001	0.001	0.000	1	5	0.923	0.044	0.002	0.923	0.525	0.525	See	See
13	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.006	0.000	1.000	0.106	0.107	See	See
14	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.001	0.001	0.000	-1	5	0.923	0.044	0.001	0.923	0.561	0.561	See	See
15	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.004	0.000	0.005	-1	5	1.000	0.006	0.000	1.000	0.820	0.821	See	See
16	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.001	0.001	0.000	-1	1	0.895	0.043	0.001	0.895	0.553	0.553	See	See
17	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	1.000	0.006	0.000	1.000	0.391	0.391	See	See
18	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.021	0.000	0.001	0.000	1	1	0.895	0.042	0.001	0.895	0.501	0.501	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.006	0.000	1.000	0.111	0.111	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.001	0.000	0.000	1	100	0.919	0.070	0.001	0.919	0.536	0.536	See	See
21	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.006	0.000	1.000	0.119	0.119	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.037	0.001	0.000	0.000	-1	100	0.919	0.067	0.001	0.919	0.556	0.556	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.001	0.000	0.002	-1	100	1.000	0.006	0.000	1.000	0.398	0.399	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.130	0.002	300	0.006	0.000	0.789	0.508	0.021	0.789	0.256	0.256	See	See
1	HistGradientBoostingClassifier_best	predict	100000	1	100	0.021	0.005	300	0.000	0.021	1.000	0.419	0.008	1.000	0.050	0.050	See	See